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Three  experiments  involved  college  students  receiving  and  following  instructions  of  various  lengths  for 
navigating  in  a  three-dimensional  space  displayed  on  a  computer  screen.  The  purpose  was  to  evaluate 
which  is  the  best  modality  for  presenting  navigation  instructions  so  that  they  can  be  executed  suc¬ 
cessfully.  Single  modalities  {read,  hear,  and  see)  were  considered  along  with  dual  modalities  presented 
simultaneously  or  successively.  It  was  found  that  when  there  were  differences  between  single  modalities, 
generally  execution  accuracy  was  best  for  see  and  worst  for  read.  Information  presented  in  two  modalities 
did  not  yield  better  accuracy  than  information  presented  twice  in  a  single  modality.  Also,  the  ordering  of 
modalities  depended  on  the  extent  of  practice.  Thus,  presentation  modality  does  not  have  a  consistently 
large  effect  on  receiving  and  following  navigation  instructions.  Repetition  and  the  amount  of  practice 
are  much  more  important  variables  than  is  presentation  modality  in  determining  how  well  navigation 
instructions  are  followed. 
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Following  navigation  instructions  is  a  common  task  in  everyday 
life  where  individuals  must  navigate  through  buildings,  neighbor¬ 
hoods,  construction  sites,  parking  structures,  and  space.  Consider, 
for  example,  getting  directions  about  where  to  find  merchandise 
in  a  large  multi-story  department  store  (e.g.,  Macy’s  or  Harrods). 
In  other,  more  serious  cases,  following  navigation  instructions  can 
have  life-or-death  consequences.  For  instance,  errors  in  communi¬ 
cation  between  air  traffic  control  (ATC)  and  pilots  can  have  severe 
repercussions.  Even  small  differences  in  the  accuracy  of  following 
navigation  instructions  could  lead  to  serious  accidents.  For  exam¬ 
ple,  if  ATC  tells  pilots  to  turn  right,  and  they  turn  left  instead, 
a  collision  might  occur  resulting  in  casualties.  Finding  ways  to 
minimize  these  communication  errors  is  thus  a  critical  question 
for  research.  One  concern  is  which  modality  would  be  best  to 
present  navigation  instructions  so  that  the  recipient  can  under¬ 
stand,  remember,  and  execute  those  instructions  with  minimal 
errors.  Messages  are  usually  presented  by  ATC  in  the  auditory 
modality,  with  pilots  hearing  the  messages  as  spoken  commands. 
ITowever,  with  current  technology  (e.g.,  data  link',  Kerns,  1991; 
Lancaster  &  Casali,  2008)  the  visual  modality  can  be  used  instead, 
with  pilots  reading  the  messages  as  written  commands.  A  third 
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possibility  also  involves  visual  presentation,  but  in  this  case  with¬ 
out  words,  with  commands  shown  as  pictures  or  symbols  (see, 
e.g.,  Tversky,  2003,  for  a  discussion  of  the  use  of  such  graphics). 
Pilots  often  see  electronic  displays  and  navigational  charts,  but  ATC 
does  not  currently  present  navigation  instructions  in  that  manner. 
Another  option  would  be  to  present  navigation  instructions  in  more 
than  one  modality  either  simultaneously  or  sequentially.  We  con¬ 
sider  each  of  these  alternatives  in  the  present  study,  in  which  we 
use  an  experimental  paradigm  simulating  a  communication  situ¬ 
ation  in  which  individuals  such  as  pilots  receive  and  then  follow 
navigation  instructions  like  those  from  ATC. 

In  the  present  study,  we  compared  the  comprehension  of  navi¬ 
gation  instructions  that  were  heard,  read,  or  seen  by  the  subjects, 
with  the  three  presentation  modes  equated  in  message  presen¬ 
tation  time  to  permit  a  pure  assessment  of  modality  effects.  In 
contrast,  in  the  actual  implementation  of  data  link,  the  visual 
presentation  is  essentially  permanent.  When  this  visual  data  link 
procedure  was  compared  to  an  auditory  procedure  (Flelleberg  & 
Wickens,  2003 ),  the  visual  mode  was  better  than  the  auditory  mode 
in  terms  of  following  navigation  instructions,  presumably  because 
of  the  differences  in  the  permanence  of  the  presentation.  However, 
a  more  recent  study  involving  data  link  by  Lancaster  and  Casali 
(2008)  found  a  disadvantage  for  the  visual  mode  relative  to  the 
auditory  mode  in  terms  of  both  increased  time  to  respond  and 
ratings  of  workload.  Furthermore,  in  a  data  link  study  McCann, 
Morrow,  Rodvold,  and  Mackintosh  (1998)  found  an  advantage  for 
the  auditory  mode  over  the  visual  mode  in  terms  of  actions  related 
to  clarification  of  navigation  messages.  Moreover,  without  the 
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differences  in  permanence  between  the  auditory  and  visual  modes, 
Wickens,  Sandry,  and  Vidulich  (1983)  found  that  pilots  receiv¬ 
ing  navigation  instructions  performed  better  with  auditory  than 
with  visual  presentation.  An  advantage  for  auditory  compared  to 
visual  presentation  modalities  has  also  been  found  in  many  stud¬ 
ies  involving  memory  for  word  lists  across  short  or  long  retention 
intervals  (e.g.,  Crowder  &  Morton,  1969;  Gardiner,  Gardiner,  & 
Gregg,  1983;  Goolkasian  &  Foos,  2002;  Murdock,  1967).  Explana¬ 
tions  for  these  modality  effects  include  Penney’s  (1989)  suggestion 
that  auditory  and  visual  items  are  processed  in  separate  streams, 
with  a  code  for  acoustic  material  longer  lasting  than  that  for  visual 
material.  Nairne’s  (1988)  feature  model  provides  an  alternative 
explanation  in  terms  of  subjects’  general  preference  for  auditory 
over  visual  features  as  recall  cues. 

When  the  material  to  be  remembered  consists  of  navigation 
instructions,  another  important  consideration  is  that  the  move¬ 
ment  space  is  visual  so  that  auditory  presentation  provides  a  mixed 
mode,  which  has  been  found  to  reduce  cognitive  load  (e.g.,  Mousavi, 
Low,  &  Sweller,  1995).  Related  to  this  finding  is  the  demonstration 
by  Brooks  ( 1 967 )  of  a  conflict  between  reading  verbal  messages  and 
imagining  the  spatial  relations  that  the  messages  describe;  such 
a  conflict  was  not  found  when  the  verbal  messages  were  heard 
rather  than  read.  Visual  messages  presenting  the  spatial  informa¬ 
tion  directly  or  with  symbols,  rather  than  with  words,  might  be 
another  way  to  avoid  this  conflict.  Such  visual  messages  would 
also  benefit  from  the  well-established  picture  superiority  effect 
(e.g..  Nelson,  Reed,  &  Walling,  1976;  Paivio  8;  Csapo,  1973;  Shepard, 
1 967 ;  Snodgrass  Si  McClure,  1975),  whereby  there  is  better  memory 
for  items  presented  as  pictures  than  as  words.  The  advantage  for 
such  visual  messages  is  also  consistent  with  the  stimulus/central¬ 
processing/response  (S-C-R)  compatibility  model  of  a  pilot’s  task 
based  on  the  assumption  that  spatial  tasks  (e.g.,  moving  in  a  space) 
are  more  compatible  with  visual  inputs  than  with  auditory  inputs 
(Wickens  &  Hollands,  2000;  Wickens,  Vidulich,  &  Sandry-Garza, 
1984). 

Many  existing  psychological  theories,  including  the  theory  of 
multiple  resources  in  cognitive  processing  (e.g.,  Wickens,  2008),  the 
dual-coding  theory  of  memory  (Paivio,  1971, 1991 ),  and  theories  of 
multimedia  learning  (Mayer,  2001;  Mayer  &  Gallini,  1990;  Mayer 
&  Sims,  1994),  predict  that  two  modalities  would  be  better  than 
one  for  learning  information.  (It  should  be  noted,  however,  that  the 
theory  of  multiple  resources  is  relevant  to  concurrent  processing  of 
the  various  resources  rather  than  to  sequential  processing  of  them, 
and  the  theories  of  multimedia  learning  do  not  predict  superior 
performance  for  two  modalities  when  those  modalities  compete 
for  attention.)  Thus,  presenting  navigation  instructions  in  more 
than  one  modality  simultaneously,  and  perhaps  also  sequentially, 
might  be  expected  to  improve  comprehension  and  memory  for 
the  instructions  relative  to  presenting  them  in  a  single  modality. 
However,  in  their  study  of  the  data  link  procedure,  Helleberg  and 
Wickens  (2003)  compared  a  redundant  condition,  in  which  both 
visual  and  auditory  information  was  presented  simultaneously,  to 
single-modality  conditions,  in  which  information  was  presented  in 
either  a  visual  or  an  auditory  format.  They  found  that  flight  task 
performance  (i.e.,  measured  as  deviations  from  ATC  instructions) 
was  best  for  the  visual  condition,  worst  for  the  auditory  condi¬ 
tion,  and  intermediate  for  the  redundant  condition.  They  attribute 
the  poor  performance  in  the  auditory  and  redundant  conditions 
to  auditory  preemption  effects,  which  interrupt  the  continuous 
visual  tasks.  In  a  related  study,  Wickens,  Goh,  Helleberg,  Horrey,  and 
Talleur  (2003)  also  found  no  advantage  in  lateral  or  vertical  track¬ 
ing  for  a  redundant  condition,  although  auditory  presentation  was 
modestly  superior  to  visual  presentation  in  that  case,  attributed  to 
the  head-down  nature  of  the  visual  scanning.  Also,  Lancaster  and 
Casali  (2008)  found  no  advantage  for  a  redundant  condition  over  a 
pure  auditory  condition  in  terms  of  time  to  respond  and  workload. 


with  both  of  those  conditions  superior  to  the  pure  visual  condition 
by  those  measures.  In  a  recent  meta-analysis,  Lu  et  al.  (in  press) 
point  out  that  although  redundant  modality  combinations  have 
traditionally  been  considered  beneficial,  there  can  be  a  cost  reflect¬ 
ing  competition  for  attentional  resources.  Their  analysis  revealed 
that  redundant  auditory-visual  tasks  were  generally  more  accurate 
but  slower  than  tasks  using  a  single  modality  (auditory  or  visual). 
They  attribute  this  pattern  of  results  to  the  fact  that  redundancy 
helps  guarantee  security  (i.e.,  redundancy  provides  more  opportu¬ 
nities  for  the  information  to  be  noticed)  but  does  so  at  the  expense 
of  efficiency.  These  earlier  studies  of  redundancy  were  restricted 
to  simultaneous  presentation  of  multiple  modalities.  Some  of  the 
costs  of  redundancy  (e.g.,  those  concerning  interruption  and  com¬ 
petition  for  attention)  would  not  accrue  when  the  modalities  are 
presented  sequentially  instead  of  simultaneously,  at  least  when 
there  is  a  control  for  the  total  duration  of  the  presentation. 

To  investigate  the  relative  benefits  of  various  modalities  and 
combinations  of  modalities  for  presenting  navigation  instructions, 
we  used  an  experimental  laboratory  paradigm  that  isolates  the  nav¬ 
igation  task  and  has  already  revealed  many  relevant  findings  (e.g., 
Schneider,  Healy,  &  Barshi,  2004;  Schneider,  Healy,  Barshi,  &  Kole, 
201 1 ).  In  our  experiments,  college  students  see  a  computer  screen 
showing  a  grid  of  four  matrices  stacked  on  top  of  each  other  and 
are  given  messages  instructing  them  to  move  in  the  grid  by  clicking 
with  a  computer  mouse  (see  Fig.  1).  This  task  is  analogous  to  the 
aviation  task  as  well  as  to  the  task  mentioned  in  the  Introduction 
of  getting  and  following  directions  about  where  to  find  merchan¬ 
dise  in  a  large  multi-story  department  store  (with  the  matrices 
corresponding  to  different  floors).  The  measure  of  performance  is 
accuracy  in  following  the  messages.  Because  of  the  large  effects  of 
message  length  on  memory  for  navigation  commands  (e.g.,  Loftus, 
Dark,  &  Williams,  1979),  the  messages  vary  substantially  in  length. 
In  some  cases,  simulating  pilot  behavior,  students  are  also  asked  to 
repeat  back  the  instructions  before  executing  them.  This  paradigm 
differs  from  the  pilot  task  in  many  important  respects  (e.g.,  expe¬ 
rienced  pilots  presumably  have  much  more  extensive  practice  and 
there  are  high  visual  demands  on  pilots).  However,  in  a  study  with 
certified  pilots  who  received  realistic  voice  ATC  navigational  and 
operational  instructions  while  flying  a  flight  simulator,  Mauro  and 
Barshi  (1999)  found  results  consistent  with  those  found  with  col¬ 
lege  students  in  the  present  task. 

In  one  experiment  (Schneider  et  al.,  2004),  we  compared  two 
groups  of  students,  an  auditory  group  receiving  auditory  messages 
(hear)  and  a  visual  group  receiving  visual  messages  shown  on  the 


Left  two  squares 
Down  two  levels 
Forward  one  step 


Fig.  1.  Sample  display  showing  a  message  with  three  commands.  The  numbers  in 
the  matrices  show  the  required  clicks:  they  were  not  shown  to  the  subjects.  The 
starting  point  is  indicated  by  the  filled-in  square. 
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side  of  the  grid  (read).  Students  received  72  different  messages 
varying  from  one  to  six  commands.  We  found  a  disadvantage  for 
reading  relative  to  hearing  commands.  However,  the  disadvantage 
for  reading  occurred  only  for  longer  messages,  perhaps  because  all 
visual  commands  were  shown  simultaneously  and  subjects  did  not 
budget  reading  time  properly. 

In  another  experiment  (Schneider  et  al.,  2011),  we  com¬ 
pared  two  groups  of  students:  a  hear  group  (verbal),  in  which 
students  listened  to  messages,  and  a  see  group  (spatial),  in 
which  the  commands  were  shown  on  the  computer  as  simu¬ 
lated  movements.  We  found  an  advantage  for  see  relative  to 
hear.  However,  there  was  more  improvement  with  practice  for 
hear  than  for  see,  so  the  advantage  for  see  was  found  only 
in  the  first  half  of  the  test,  with  no  difference  in  the  last 
half. 

The  present  study  expands  on  these  earlier  investigations.  It 
compares  hear,  read,  and  see  conditions  on  their  own  and  in  com¬ 
bination,  with  two  different  types  of  see  conditions,  one  in  which 
subjects  see  response  locations  and  the  other  in  which  they  see 
symbols  representing  the  movements.  Specifically,  Experiment  1 
compares  hear  and  read  conditions  to  a  condition  including  both 
hearing  and  reading  simultaneously.  However,  the  both  condition 
allows  subjects  to  read  the  messages  at  a  rate  different  from  that  at 
which  they  are  heard,  which  could  cause  an  asynchrony  in  tim¬ 
ing,  thereby  making  it  difficult  to  find  the  expected  advantage 
for  message  redundancy.  To  eliminate  this  problem  and  thereby 
enhance  the  ability  to  find  an  advantage  for  combining  modali¬ 
ties,  Experiment  2  involves  sequential,  rather  than  simultaneous 
presentation  of  two  modalities.  In  addition.  Experiment  2  adds 
a  see  condition,  in  which  response  locations  are  shown  succes¬ 
sively  on  the  grid.  Each  message  is  repeated  either  in  the  same 
modality  (read-read,  hear-hear,  see-see)  or  in  two  different  modal¬ 
ities  (read-hear,  hear-read,  read-see,  see-read,  hear-see,  see-hear). 
The  see  condition  of  Experiment  2  differs  from  the  read  condition 
in  the  location  where  the  information  is  presented  (within  or  to 
the  right  of  the  grid).  To  eliminate  differences  between  see  and 
read  conditions  in  terms  of  stimulus  location  and  thereby  allow 
for  a  pure  comparison  of  nonverbal  and  verbal  stimuli  and,  thus,  a 
more  controlled  test  of  the  picture  superiority  effect.  Experiment 
3  introduces  a  see  condition  involving  arrow  symbols  shown  at 
the  side  of  the  grid  and  contrasts  it  to  a  read  condition,  in  which 
words  instead  are  shown  at  the  side  of  the  grid.  Single  condi¬ 
tions,  in  which  a  message  is  shown  only  once  (see,  read),  double 
conditions,  in  which  a  message  is  repeated  in  the  same  modal¬ 
ity  (see-see,  read-read),  and  mixed  conditions,  in  which  a  message 
is  repeated  in  both  modalities  (see-read,  read-see),  are  all  exam¬ 
ined. 

1.  Experiment  1 

In  Experiment  1,  three  different  presentation  conditions  were 
compared  with  all  subjects  exposed  to  all  conditions:  hear,  read,  and 
both  hear  and  read  simultaneously.  The  conditions  were  admin¬ 
istered  in  three  different  orders  such  that  across  subjects  each 
condition  occurred  in  each  test  position. 

3.3.  Method 

3.3.3.  Subjects 

Thirty-six  native  English  speaking  college-age  adults  partic¬ 
ipated  for  payment  ($10/h)  (27  subjects)  or  for  introductory 
psychology  course  credit  (9  subjects). 

3.3.2.  Design  and  materials 

The  study  implemented  a  3  (message  modality)  x  3  (message 
modality  order)  mixed  factorial  design.  Message  modality  was  a 


within-subject  variable;  message  modality  order  was  a  between- 
subjects  variable,  with  12  subjects  in  each  order.  The  dependent 
measure  was  proportion  of  correct  manual  movement  responses 
scored  on  an  all-or-none  basis  ignoring  the  correctness  of  the  oral 
repetition  responses. 

Subjects  saw  a  computer  screen  with  four  4x4  matrices  stacked 
vertically.  Subjects  either  heard  messages  telling  them  where  to 
move  on  the  grid  (hear),  read  messages  (read),  or  both  heard  and 
read  messages  simultaneously  (both).  These  messages  were  devel¬ 
oped  by  Barshi  and  Healy  (2002)  and  were  spoken  by  a  male  native 
English  speaker.  The  read  messages  were  equated  (as  closely  as 
possible  given  software  constraints)  to  the  hear  messages  in  terms 
of  total  presentation  time.  Message  length  ranged  from  one  to  six 
commands,  which  instructed  subjects  to  turn  left  or  right  a  given 
number  of  squares,  climb  up  or  down  a  given  number  of  levels, 
or  move  forward  or  back  a  given  number  of  steps.  The  duration 
of  each  specific  spoken  command  depended  on  the  exact  words 
used,  so  the  duration  differed  across  commands,  although  it  was 
fixed  for  a  given  command.  Because  the  commands  were  limited  to 
navigation  in  three  dimensions,  message  lengths  longer  than  three 
commands  had  to  repeat  dimensions.  The  command  dimensions 
were  always  presented  in  the  same  order  (right/left,  up/down,  for¬ 
ward/back)  because  commands  to  pilots  are  also  always  given  in  a 
consistent  order. 

Subjects  received  24  trials  in  each  of  the  three  modalities,  with 
four  of  each  message  length  (one  to  six  commands)  in  each  modal¬ 
ity.  These  message  lengths  were  presented  in  a  pseudorandom 
order,  such  that  each  block  of  12  trials  consisted  of  two  trials  of 
each  length. 

The  conditions  were  ordered  according  to  a  Latin  Square,  with 
one-third  of  the  subjects  receiving  the  set  of  read  messages  (R)  first, 
then  fiear(H),  then  both  (B),  one-third  HBR,  and  one-third  BRH.  The 
order  of  the  messages  was  the  same  for  all  subjects;  for  example. 
Message  1  always  consisted  of  the  same  information  regardless  of 
modality. 

Each  command  in  the  messages  contained  four  words  (e.g.,  “turn 
left  two  squares”).  All  subjects  were  to  repeat  back  the  messages, 
as  pilots  do  when  receiving  ATC  instructions.  In  the  read  condition, 
a  message  appeared  on  the  screen  with  all  the  commands  shown 
simultaneously,  with  one  command  presented  per  line  to  the  right 
of  the  complete  empty  grid  of  four  4x4  matrices.  In  the  hear  condi¬ 
tion,  subjects  heard  the  commands  as  they  viewed  the  empty  grid, 
and  in  the  both  condition,  subjects  simultaneously  heard  the  com¬ 
mands  and  saw  them  on  the  right  of  the  grid.  The  complete  grid  was 
visible  on  the  screen  as  the  subjects  received  the  messages,  and  it 
remained  visible  as  the  subjects  responded  to  the  messages  in  each 
condition. 


3.3.3.  Procedure 

Subjects  were  given  six  preliminary  practice  trials  before  the 
experimental  trials  in  the  first  modality,  three  in  the  second,  and 
three  in  the  third. 

Subjects  were  given  a  message  in  one  of  the  modality  conditions 
specifying  where  to  move  on  the  grid;  they  were  to  repeat  this  mes¬ 
sage  and  then  click  a  button  marked  “Done  speaking.”  At  this  point 
the  starting  square  on  the  grid  became  highlighted  (i.e.,  it  was  filled 
rather  than  empty);  the  starting  square  for  the  preliminary  practice 
trials  is  shown  in  Fig.  1 ;  the  starting  square  for  the  experimen¬ 
tal  trials  was  the  mirror  image  on  each  dimension.  Next,  subjects 
followed  the  directions  by  clicking  on  the  squares.  Each  square  in 
the  grid  became  highlighted  once  it  was  clicked  and  stayed  high¬ 
lighted  until  the  next  square  was  clicked  and  highlighted.  Thus, 
only  one  square  was  highlighted  at  a  time.  When  finished,  sub¬ 
jects  clicked  “Done  clicking,”  and  the  next  trial  began  after  a  1-s 
delay. 
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3.2.  Results 

The  proportion  of  correct  responses  was  equivalent  in  the 
read  (M  =  .477,  SEM=.029),  hear  (M=.491,  S£M  =  .028).  and  both 
(3W=.488,  S£M  =  . 029)  conditions.  The  main  effect  of  modality  was 
not  significant,  F(2,  66)<1;  however,  modality  did  interact  sig¬ 
nificantly  with  modality  order,  F(4,  66)  =  6.1 25,  p<. 001,  rp-  =  .27\, 
because  accuracy  Increased  from  the  first  (M  =  .444,  S£M  =  . 029)  to 
the  second  (3Vf  =  .478,  S£M  =  . 029)  to  the  third  (M  =  .534,  S£M  =  . 027) 
modality  position.  Thus,  performance  improved  with  practice 
regardless  of  modality. 

3.3.  Discussion 

The  combination  of  modalities  (both  condition)  did  not  yield 
higher  performance  than  each  modality  presented  separately  (read 
or  hear  condition),  perhaps  because  of  the  asynchrony  between  the 
timing  of  reading  a  given  command  and  the  timing  of  hearing  the 
same  command  for  the  combined  modalities. 

Neither  hearing  alone  nor  reading  and  hearing  together  elevated 
performance  over  reading  alone.  Instead,  performance  improved 
with  practice:  performance  was  worst  for  the  first  condition  tested 
and  best  for  the  last  condition  tested. 

2.  Experiment  2 

in  Experiment  1,  subjects  In  the  read  condition  might  have 
read  the  stimuli  at  a  rate  different  from  that  in  which  the  stimuli 
were  presented  in  the  hear  condition,  despite  the  equivalent  pre¬ 
sentation  times.  Thus,  an  advantage  for  the  dual-mode  condition 
might  be  evident  only  if  the  two  modes  are  presented  separately  in 
time.  Hence,  in  Experiment  2  sequential  (rather  than  simultaneous) 
dual-mode  conditions  were  compared  to  single-mode  conditions. 
To  provide  an  appropriate  baseline  in  terms  of  message  repeti¬ 
tion,  the  single-mode  conditions  involved  repetition  of  the  same 
stimuli  in  the  same  modality.  Also,  a  see  condition  was  added  to 
the  hear  and  read  conditions.  The  comparison  of  conditions  was 
made  between  subjects,  rather  than  within  subjects  as  in  Experi¬ 
ment  1.  The  between-subjects  manipulation  also  allowed  for  equal 
amounts  of  practice  in  each  modality,  so  that  the  amount  of  practice 
could  not  mask  any  modality  effects. 

2.3.  Method 

2.1.1.  Subjects 

Seventy-two  native  English  speaking  college  students  partici¬ 
pated  for  introductory  psychology  course  credit. 

2.3.2.  Design  and  materials 

The  study  implemented  a  6  (block  of  trials)  x  9  (message  modal¬ 
ity  order)  mixed  factorial  design.  Block  of  trials  was  a  within-subject 
variable:  message  modality  order  was  a  between-subjects  variable, 
with  6  or  12  subjects  in  each  modality  order.  The  dependent  mea¬ 
sure  was  again  proportion  of  correct  responses. 

Subjects  saw  the  same  computer  screen  as  in  Experiment  1 .  They 
either  heard  messages  that  told  them  where  to  move  on  the  grid 
(hear),  read  messages  (read),  or  saw  the  required  movements  on 
the  grid  (see).  In  the  see  condition,  as  in  Schneider  et  al.  (201 1 ),  each 
square  was  highlighted  on  the  grid  in  the  same  order  that  subjects 
were  to  click  when  following  the  read  or  hear  messages.  The  time 
to  see  the  movements  in  the  see  condition  was  equated  to  the  time 
to  hear  the  messages  in  the  hear  condition.  The  messages  in  the 
read  condition  were  shown  one  command  at  a  time  (rather  than 
showing  all  commands  simultaneously  as  in  Experiment  1),  with 
the  duration  of  each  command  also  equated  to  that  in  the  hear  con¬ 
dition.  Each  command  was  presented  on  a  single  line  to  the  right  of 


the  grid.  A  given  command  was  presented  and  then  removed  before 
the  next  command  appeared,  and  successive  commands  appeared 
on  different  lines,  with  each  command  appearing  on  a  line  below 
the  previous  command. 

The  set  of  commands  used  were  those  employed  by  Schneider 
et  al.  (2011),  who  compared  see  and  hear  conditions.  All  read  and 
hear  commands  included  only  three  words  (e.g.,  left  two  squares). 
Instead  of  presenting  commands  in  a  fixed  order,  this  set  of  com¬ 
mands  was  constructed  so  that  the  number  of  movements  along 
each  movement  dimension  (right/left,  up/down,  forward/back) 
was  equated  at  each  serial  position  of  the  commands. 

Subjects  received  72  trials,  divided  into  six  blocks  of  12,  includ¬ 
ing  two  of  each  of  the  six  message  lengths  (one  to  six  commands) 
in  a  pseudorandom  order. 

There  were  three  single-mode  conditions  (read-read,  hear-hear, 
and  see-see),  in  which  a  given  message  was  presented  twice  in  suc¬ 
cession  in  the  same  modality,  with  12  subjects  in  each  condition. 
There  were  six  dual-mode  conditions  (read-hear,  read-see,  hear- 
read,  hear-see,  see-read,  and  see-hear),  in  which  a  given  message  was 
presented  in  one  modality  and  then  repeated  in  another  modality, 
with  6  subjects  in  each  condition.  The  messages  were  repeated  in 
the  single-mode  conditions  so  that  performance  could  be  appro¬ 
priately  compared  to  that  In  the  dual-mode  conditions,  where  the 
messages  were  necessarily  repeated.  The  order  of  the  72  trials  was 
the  same  for  each  of  the  nine  conditions. 

2.3.3.  Procedure 

The  same  procedure  was  used  as  in  Experiment  1  except  that 
subjects  did  not  repeat  back  the  messages  or  see  any  button  marked 
“Done  speaking”  because  oral  repetition  responses  are  not  appro¬ 
priate  for  the  see  condition.  At  the  start  of  the  session,  subjects 
were  given  six  preliminary  practice  trials  for  their  specific  condi¬ 
tion.  Also,  the  starting  square  was  denoted  with  a  red  asterisk  and 
shown  at  the  start  of  each  trial. 

2.2.  Results 

Two  analyses  were  conducted:  the  first,  restricted  to  the  single¬ 
mode  conditions,  allows  for  the  assessment  of  the  relative  merits  of 
each  modality  on  its  own.  The  second,  an  overall  analysis  including 
all  nine  conditions,  allows  for  a  comparison  of  single-mode  and 
dual-mode  conditions. 

2.2.3.  Single-mode  conditions  only 

The  proportion  correct  was  highest  in  see-see  (3V3  =  .804, 
SEM  =  .015),  lowest  in  read-read  (M  =  .684,  S£3V3=  .01 9),  and  interme¬ 
diate  in  hear- hear  (M= .756,  SEM =.017).  The  main  effect  of  modality 
was  significant  in  the  analysis  restricted  to  the  single-mode  condi¬ 
tions,  F(2, 33  )=  4.71 0,  p  =  .01 6,  =  .222.  Fisher’s  PESO  post  hoc  tests 
indicated  that  only  the  difference  between  see-see  and  read-read 
was  significant.  There  was  also  a  significant  interaction  of  modality 
and  block  (see  Fig.  2),  £(10, 165)  =  1.947,  p  =  . 042,  r/^  =  .106,  because 
performance  on  see-see  was  at  least  somewhat  better  than  that  on 
hear-hear  in  all  but  the  last  block,  in  agreement  with  Schneider  et  al. 
(201 1 ),  and  read-read  was  worse  than  the  other  two  conditions  on 
all  but  the  first  block,  where  it  was  somewhat  better  than  hear-hear. 

2.2.2.  All  conditions 

When  all  conditions  were  considered,  see-see,  see-read,  and 
hear-read  yielded  the  best  performance,  and  read-read  the  worst 
(see  Fig.  3  for  the  means  and  standard  errors  in  each  condition).  The 
analysis  including  all  conditions  yielded  a  significant  main  effect  of 
condition,  £(8, 63)  =  2.119,p  =  .047,  =  .212.  Fisher’s  PLSD  post  hoc 

tests  indicated  that  along  with  the  significant  difference  between 
see-see  and  read-read,  there  were  significant  differences  between 
see-read  and  read-read  and  between  hear-read  and  read-read.  Thus, 


196 


A.F.  Healy  et  a\./  Journal  of  Applied  Research  in  Memory  and  Cognition  2  (2013)  192-199 


Single-Mode  Conditions  Only 


Block 

Fig.  2.  Proportion  of  correct  responses  in  Experiment  2  as  a  function  of  modality 
and  block  for  the  single-mode  conditions  only.  Note.  RR= read-read,  HH  =  hear-hear, 
SS  =  see-see.  Error  bars  represent  standard  errors  of  the  mean. 

reading  alone  was  not  conducive  to  accurate  performance,  but 
reading  preceded  by  either  hearing  or  seeing  was. 

2.3.  Discussion 

Reading  after  either  of  the  other  two  modes  (i.e.,  hear-read  or 
see-read)  and  see-see  were  best,  whereas  read-read  was  worst.  Find¬ 
ing  that  see-see  was  one  of  the  best  conditions  implies  that  words 
are  more  difficult  to  encode  than  moves.  Also,  finding  that  read-read 
was  the  worst  condition  implies  that  reading  is  worse  than  hearing 
or  seeing.  Importantly,  reading  was  beneficial  only  when  preceded 
by  the  hearing  or  seeing  modes. 

Why  is  the  read  mode  particularly  helpful  when  it  follows 
another  mode?  The  read  mode  requires  looking  at  the  words  on 
the  side  of  the  grid  rather  than  where  the  moves  are  made  in  the 
grid.  Therefore,  the  read  mode  might  interfere  with  following 


Condition 

Fig.  3.  Proportion  of  correct  responses  in  Experiment  2  as  a  function  of  condi¬ 
tion.  Note.  RR  =  read-read,  HH  =  hear-hear,  SS  =  see-see,  HS  =  hear-see,  SH  =  see-hear, 
RS  =  read-see,  SR  =  see-read,  RH  =  read-hear,  HR = hear-read.  The  single-mode  con¬ 
ditions  are  separated  from  the  dual-mode  conditions  by  a  vertical  line,  and  both 
sets  of  conditions  are  ordered  from  lowest  to  highest  accuracy.  Error  bars  represent 
standard  errors  of  the  mean. 


the  moves  in  the  grid  if  subjects  look  at  or  point  to  each  successive 
location  as  they  receive  the  commands  (Brooks,  1967).  Thus,  one 
possibility  is  that  subjects  follow  the  moves  in  the  grid  on  the  first 
presentation  but  do  not  follow  the  moves  in  the  grid  on  the  second, 
so  the  read  mode  does  not  interfere  after  presentation  in  a  different 
mode  and  actually  might  serve  as  a  valuable  confirmation  of  what 
was  already  seen  or  heard.  This  explanation  is  consistent  with  the 
proposal  by  Lu  et  al.  (in  press)  that  redundancy  helps  guarantee 
security. 

Contrary  to  the  dual-coding  theory  of  memory  (Paivio,  1971) 
and  theories  of  multimedia  learning  (e.g.,  Mayer,  2001),  these 
results  showed  that  two  modalities  are  not  always  better  than  one 
for  processing  information  (see  Lee  &  Bowers,  1997,  for  a  similar 
finding  in  multimedia  learning). 

3.  Experiments 

The  see  condition  used  in  Experiment  2  (and  in  Schneider  et  al., 
2011)  displayed  every  move  on  the  computer  screen  as  subjects 
would  make  the  moves  themselves.  This  procedure  might  have 
given  an  unfair  advantage  to  the  see  condition  and  also  made  the 
read  and  see  conditions  differ  in  other  ways  than  those  involving 
verbal  or  nonverbal  presentation  of  the  messages.  Specifically,  the 
read  and  see  conditions  differed  in  terms  of  the  location  of  the 
stimuli  (to  the  right  of  the  grid  or  within  the  grid,  respectively). 
Stimuli  on  the  side  of  the  grid  might  require  scanning  back  and 
forth  from  the  stimuli  to  the  grid  if  subjects  follow  the  movements 
in  the  grid  as  they  receive  the  navigation  commands.  To  investigate 
differences  between  verbal  and  nonverbal  messages  without  any 
irrelevant,  confounding  differences  between  conditions,  in  Experi¬ 
ment  3  we  employed  a  see  condition  that  used  symbols  (arrows)  to 
convey  the  direction  and  magnitude  of  the  required  movements, 
with  the  arrows  occurring  to  the  right  of  the  grid  just  like  the 
words  in  the  read  condition.  According  to  earlier  research  exam¬ 
ining  spatial  compatibility  effects  (e.g..  Miles  &  Proctor,  2012), 
arrow  symbols  should  be  more  compatible  than  words  with  spatial 
locations,  and  thus,  the  see  condition  might  be  expected  to  lead 
to  better  execution  performance  than  the  read  condition  in  the 
present  spatial  navigation  task.  In  this  case,  we  examined  see  and 
read  conditions  with  both  a  single  presentation  (see  and  read)  and 
two  repeated  presentations  (see-see  and  read-read)  as  well  as  com¬ 
bined  conditions  (see-read  and  read-see).  No  hear  conditions  were 
included. 

3.1.  Method 

3.1.1.  Subjects,  design,  materials,  and  procedure 

The  subjects,  design,  materials,  and  procedure  for  Experiment  3 
were  equivalent  to  those  in  Experiment  2,  except  as  noted  here. 

The  study  employed  a  6  (block  of  trials)  x  3  (presentation  type) 
mixed  factorial  design.  Again,  the  dependent  measure  was  pro¬ 
portion  of  correct  responses.  Block  of  trials  was  a  within-subject 
variable;  presentation  type  was  a  between-subjects  variable.  Pre¬ 
sentation  type  included  single  (read  or  see)  versus  double  (read-read 
or  see-see)  versus  mixed  (read-see  or  see-read)  presentation  of  mes¬ 
sages,  with  1 2  subjects  in  each  of  the  six  specific  presentation  types. 

The  read  commands  were  presented  in  the  same  way  as  in  the 
read  condition  of  Experiment  2  (see  Eig.  1 ).  The  see  commands  were 
presented  as  colored  arrows  and  boxes  (see  Fig.  4).  After  the  ini¬ 
tial  command  appeared,  each  subsequent  command  was  timed  to 
appear  on  the  screen  after  1 .25  s. 

3.2.  Results 

Two  analyses  were  conducted,  an  overall  analysis  including  all 
three  presentation  types  (single,  double,  or  mixed)  and  a  second 
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Fig.  4.  Sample  display  showing  a  message  with  three  commands  in  the  see  modality 
of  Experiment  3.  The  numbers  in  the  matrices  show  the  required  clicks;  they  were 
not  shown  to  the  subjects.  The  starting  point  is  indicated  by  the  filled-in  square. 


analysis  excluding  the  mixed  condition  but  including  the  variable  of 
modality  {read  or  see).  Two  separate  analyses  were  needed  because 
the  variables  of  modality  and  presentation  type  were  not  fully 
crossed.  The  analysis  excluding  the  mixed  group  allowed  modal¬ 
ity  {read  versus  see)  to  be  crossed  with  presentation  type  (single 
versus  double  presentation). 

3.2.  J.  Overall  ANOVA 

Performance  was  better  for  double  presentation  (M  =  .715, 
SEM=. 013)  and  mixed  presentation  (M  =  .644,  S£M=. 014),  when 
messages  were  viewed  twice,  than  for  single  presentation  (M  =  .542, 
S£'M  =  . 01 4).  The  main  effect  of  presentation  type  was  significant, 
f(2,  69)  =  11.317,  p<.001,  rp-  =  .2Al.  Fisher’s  PLSD  post  hoc  tests 
revealed  that  although  double  was  higher  than  mixed  presenta¬ 
tion,  the  difference  was  only  marginally  significant  (p  =  .056),  but 
the  advantage  for  twice-viewed  messages  over  single  presentation 
held  when  the  messages  were  viewed  in  one  modality  (p  <  .001 )  or 
in  mixed  modalities  (p  =  .007). 


3.2.2.  ANOVA  with  modality  variable 

Performance  for  doubly  presented  messages  (M  =  .715, 
S£M=.013)  was  superior  to  that  for  singly  presented  mes¬ 
sages  (M  =  .542,  S£iW  =  .014),  F(l,  44)  =  25.235,  p<.001,  r]^  =  .364. 
Performance  on  the  read  (iW=.634,  S£M  =  .014)  and  see  (M  =  .623, 
S£M=.014)  modalities  was  virtually  equivalent,  £(1,  44)<1,  and 
did  not  depend  on  presentation  type,  F(l,  44)  =  2.373,  p  =  .131, 
p^  =  .051.  However,  subjects  improved  more  consistently  across 
blocks  for  read  than  for  see,  so  that  by  the  last  two  blocks  there 
was  actually  a  numerical  advantage  for  read  relative  to  see  (see 
Fig.  5);  the  interaction  between  modality  {read  and  see)  and  block 
was  significant,  F(5,  220)  =  2.545,  p  =  .029,  =  .055. 


Block 

Fig.  5.  Proportion  of  correct  responses  in  Experiment  3  as  a  function  of  modality  and 
block.  Note  that  the  see  condition  in  Experiment  3  involved  arrow  symbols.  Error 
bars  represent  standard  errors  of  the  mean. 

3.3.  Discussion 

Repeating  messages  in  either  modality  {read  or  see)  aided  per¬ 
formance  relative  to  presenting  messages  only  once.  However, 
repeating  messages  in  two  different  modalities  was  no  better  than 
repeating  them  in  a  single  modality,  which  is  consistent  with  the 
proposal  by  Lu  et  al.  (in  press)  that  redundancy  is  valuable  because 
it  helps  guarantee  security  if  repetition  in  the  same  modality  is  at 
least  as  effective  as  repetition  in  two  different  modalities  for  guar¬ 
anteeing  security.  In  addition,  performance  did  not  differ  for  the 
read  and  see  conditions  despite  the  better  spatial  compatibility  for 
the  arrow  symbols  used  in  the  see  condition  than  for  the  words 
used  in  the  read  condition  (Miles  &  Proctor,  2012).  Thus,  the  see 
condition  in  Experiment  3  does  not  seem  to  be  as  effective  as  the 
see  condition  in  Experiment  2,  which  displayed  every  move  on  the 
computer  screen  as  subjects  would  make  the  moves  themselves 
and  which  did  yield  significantly  better  performance  than  the  read 
condition  in  that  experiment.  As  speculated  earlier,  the  procedure 
used  for  the  see  condition  in  Experiment  2  might  have  given  it  an 
unfair  advantage  because  subjects  in  that  condition  did  not  have  to 
scan  between  the  commands  on  the  side  of  the  grid  and  the  grid 
itself. 

Also,  in  agreement  with  the  earlier  findings  of  Schneider  et  al. 
(2011),  we  found  relatively  little  improvement  with  practice  for 
the  see  modality,  although  performance  did  improve  for  the  read 
modality.  Thus,  there  was  an  interaction,  with  performance  on  the 
see  modality  somewhat  better  than  on  the  read  modality  at  the 
start  of  practice,  but  the  opposite  pattern  at  the  end  of  practice;  the 
modality  ordering  depended  on  the  amount  of  practice.  We  specu¬ 
late  that  this  difference  between  the  see  and  read  conditions  in  the 
improvement  due  to  practice  could  be  a  function  of  the  difference 
between  those  conditions  in  spatial  compatibility.  Because  of  the 
greater  spatial  compatibility  of  the  symbols  used  in  the  see  condi¬ 
tion  than  of  the  words  used  in  the  read  condition,  the  see  condition 
might  require  less  skill  development  for  the  present  task  involving 
navigation  to  locations  in  space  than  would  the  read  condition. 

4.  General  discussion 

The  present  study  assessed  which  modality  maximizes  execu¬ 
tion  performance  on  following  navigation  instructions.  The  results 
generally  show  that  the  modalities  are  ordered  from  least  to  most 
effective  as  read,  hear,  see,  but  this  ordering  was  not  found  in  all 
cases.  In  Experiment  2,  when  there  were  significant  differences 
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observed  between  see,  hear,  and  read,  they  were  ordered  such 
that  see  was  best  and  read  was  worst.  However,  as  made  clear  in 
Experiment  1 ,  practice  has  an  effect  that  overwhelms  modality  of 
presentation,  and  in  Experiment  2  the  effects  of  modality  depended 
on  practice  block.  Likewise,  in  Experiment  3  the  effect  of  practice 
was  greater  for  the  read  modality  than  for  the  see  modality,  so  the 
ordering  of  the  read  and  see  modalities  depended  on  the  amount  of 
practice. 

As  in  previous  studies  (e.g.,  Crowder  &  Morton,  1969;  Gardiner 
et  al.,  1983;  Goolkasian  &  Foos,  2002;  Murdock,  1967),  we  tended 
to  find  an  advantage  for  hear  relative  to  read.  We  can  explain 
this  difference  using  the  same  theoretical  proposals  (e.g.,  Nairne, 
1988;  Penney,  1989)  advanced  for  explaining  the  similar  findings 
in  the  standard  memory  tasks  of  an  advantage  for  auditory  rela¬ 
tive  to  visual  presentation.  However,  we  also  showed  that  visual 
presentation  tends  to  be  better  than  auditory  presentation  when 
the  visual  presentation  involves  see  rather  than  read.  The  previous 
findings  all  involved  memory  for  word  lists,  whereas  the  present 
findings  involve  memory  for  navigation  instructions,  where  there 
was  expected  to  be  a  conflict  between  reading  the  verbal  messages 
and  imagining  the  spatial  relations  that  the  messages  describe 
(Brooks,  1967).  Such  a  conflict  is  diminished  when  the  visual  mes¬ 
sages  are  seen  as  movements  or  symbols  rather  than  read  as 
words,  and  messages  seen  as  movements  or  symbols  might  benefit 
from  the  well-established  picture  superiority  effect  (Nelson  et  al., 
1976;  Paivio  &  Csapo,  1973;  Shepard,  1967;  Snodgrass  8i  McClure, 
1975)  and  from  stimulus/central-processing/response  compatibil¬ 
ity  given  the  spatial  nature  of  the  required  movements  (Wickens 
&  Hollands,  2000;  Wickens  et  al.,  1984).  The  present  findings  are 
in  agreement  with  the  earlier  studies  using  the  present  paradigm, 
which  also  showed  an  advantage  for  hear  relative  to  read  (Schneider 
et  al.,  2004)  and  for  see  relative  to  hear  (Schneider  et  al„  201 1 ). 

Consistent  with  prior  theory  (e.g.,  Mayer,  2001;  Paivio,  1971; 
Wickens,  2008),  two  different  modalities  should  have  an  advan¬ 
tage  over  a  single  modality  at  least  when  the  two  modalities  are 
presented  simultaneously  (the  theory  of  multiple  resources  applies 
only  to  concurrent,  not  sequential,  processing).  However,  no  advan¬ 
tage  for  redundant  simultaneous  dual  modalities  was  found  in 
Experiment  1  (in  accordance  with  the  findings  of  Helleberg  & 
Wickens,  2003;  Lancaster  &  Casali,  2008;  Wickens  et  al.,  2003), 
and,  when  controlling  for  the  number  of  repetitions  in  Experiments 
2  and  3,  no  advantage  for  dual  modalities  was  evident  when  the 
modalities  were  presented  sequentially.  These  findings  are  consis¬ 
tent  with  the  proposal  by  Lu  et  al.  (in  press)  that  redundancy  aids 
performance  by  guaranteeing  security  if  it  is  also  the  case  that  rep¬ 
etition  in  a  single  modality  is  at  least  as  good  as  repetition  in  two 
different  modalities  in  terms  of  guaranteeing  security. 

4.1.  Practical  application 

The  present  results  make  it  clear  that  the  ordering  of  modal¬ 
ities  for  optimal  presentation  of  navigation  instructions  depends 
on  factors  such  as  practice.  In  fact,  the  present  study  demon¬ 
strates  that  repetition  and  the  amount  of  practice  are  much  more 
important  variables  than  is  presentation  modality  for  maximizing 
performance  at  receiving  and  following  navigation  instructions  like 
those  received  by  pilots  from  ATC.  As  mentioned  earlier,  however, 
experienced  pilots  presumably  have  much  more  extensive  practice 
than  was  provided  in  the  present  experiments,  and  there  are  high 
visual  demands  on  pilots,  especially  during  takeoff  and  landing. 
Thus,  future  research  on  this  topic  should  include  longer  practice 
as  well  as  other  aspects  of  the  task  facing  pilots  but  missing  from  the 
present  paradigm.  Following  navigation  instructions,  though,  is  an 
everyday  task  not  limited  to  pilots  (e.g.,  as  mentioned  in  the  Intro¬ 
duction,  finding  merchandise  in  a  multi-story  department  store), 
and  individuals  might  be  much  less  practiced  and  might  not  have 


high  visual  demands  in  those  tasks.  Thus,  the  lessons  learned  in 
this  study  might  be  applicable  to  many  real-world  situations  in 
which  individuals  receive  and  execute  instructions  about  where 
to  make  movements  in  an  environment.  Specifically,  to  maximize 
performance,  individuals  should  be  given  sufficient  practice  at  the 
navigation  task,  and  the  navigation  messages  should  be  presented 
twice  in  a  row.  However,  the  modality  in  which  the  messages  are 
presented  (e.g.,  in  written  or  spoken  form)  is  not  crucial,  so  that  the 
most  convenient  modality  can  be  employed. 
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